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ABSTRACT 



An arts-based supplemental education program was evaluated 
qualitatively and quantitatively. It was hypothesized that this classroom 
program would cause gains in critical thinking skills, as measured through 
writing ability. The program provided 87 teachers in grades 3 through 12 with 
a classroom set of "Arts Indiana" magazine and an accompanying study guide. 
The qualitative study of teacher use of the resources and materials of the 
program identified teachers who were using the program effectively. About 
half of the program teachers agreed to participate in the quantitative part 
of the study, designed to evaluate students' critical thinking and writing 
skills. This study was completed in the context of a new state high school 
achievement test of writing. About 2,000 pretests were obtained after summer 
vacation before students participated in the supplemental program and about 
1,700 posttests of student writing were also completed. Classwide losses in 
critical thinking skills were actually seen from pretest to posttest. 

Although the supplemental resource did not cause greater gains in critical 
thinking among its users, its effective use did reduce the classwide losses 
in students ' critical thinking and writing experienced between the pretest 
and the posttest. Implications for teaching in general and for this program 
in particular are discussed. (Contains seven graphs.) (SLD) 
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The Study 

This two-year study was designed to have a qualitative component and a 
quantitative component. During the first year of the study, in the 1995-1996 
school year, 1 traveled 8,000 miles through 68 Indiana counties to visit, interview, 
and observe 87 teachers who were in a classroom program called Arts Indiana 
Magazine in the Classroom. 

The supplemental program provided program teachers with a classroom set of 
Arts Indiana magazines and an accompanying Arts Now!!! study guide. The program 
began five years earlier with four teachers and had grown to 87 teachers receiving 
approximately 3,000 magazines, monthly, by the time of this study. 

Teachers in grades third through twelfth used the supplemental resource 
especially in subjects like visual arts, language arts, humanities, and the 
performing arts. The program was designed to increase student performance 
on standardized achievement tests by inciting enhanced critical thinking skills, 
through the visual, performing, and literary arts. 

I interviewed all program teachers and transcribed taped sessions to help me 
determine who seemed to be using the resource most effectively. This led to 
my dividing the program teachers into two groups of teachers for the subsequent 
quantitative portion in year two of the study ('96-97). One group consisted of 
teachers who used the resource most effectively and the second group of program 
teachers were those who were judged to be using the resource less effectively. 
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About half of the 87 program teachers agreed to participate in the quantitative 
portion of the study. The critical path of the quantitative portion of the study 
soon became the assemblage of control groups for the study. Two control 
groups were designed. One group consisted of teachers who were interested in 
eventually receiving the free supplemental resource beyond the life of the study and 
a second control group would consist of teachers who were simply interested in 
having students participate in a writing study that measured their students’ critical 
thinking skills. 

At this same time, the Indiana Department of Education announced that the 
next high school achievement test would include a writing portion on the test that 
would resemble the writing exercise in the study. The validated writing instrument 
for the study was an NAEP instrument used in a 1992 national writing study. The 
particular instrument measured critical thinking skills through asking students to 
write about an invention. Students were asked to draft a letter to the United States 
patent office describing some sort of an invention. 

As a result, the State's sixth largest school district decided to have all tenth grade 
students participate in the study, along with teachers across grades fifth through 
twelfth; across subjects, from all parts of Indiana. Treatment and control groups 
were built and organized and I then began to deal with the new testing and privacy 
laws that had been established by the Indiana legislature. The parents or guardians of 
all students who might be in the study had to consent to their child being in the 
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study. With the new laws, no scholarly research will feel the enjoyment of having 
all students participate in a study. While many students' parents and guardians did 
sign consent papers to allow their children to be in the study, some did not. I 
believe these new laws that are supposed to protect the privacy of Indiana students 
actually harm educational research by excluding some students from studies like 
this. 

Pretest packets were sent to the 90 treatment and control group teachers with 
approximately 2,050 tests and 4,200 consent forms included. All packets had to 
be hand stamped because of the TWA airline crash and heightened security at the 
post office. Teachers were carefully prompted to give the writing tests as early 
as possible in the semester. They would give a posttest with the same NAEP prompt 
as late as possible in the semester. 

About 2,000 pretests were returned and scored using the rubric provided by the 
NAEP. Compensated interraters randomly pulled and audited ten-percent of all 
tests and the reliability rate of both the pretests and posttests were consistent and 
above the rates of the 1992 NAEP national study. 

Posttest packets were prepared and delivered prior to the end of the semester. 
About 1700 posttests were completed and returned. They were scored and 
interrated in the same fashion as the pretests. 

Using an SPSS software package, I was able to enter data on my microcomputer 
at my home office. This advance in technology saved me a considerable amount of 
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time. Upon completion of all data entry, I had the privilege of having Dr. Kim 
Metcalf help me run the data at the Smith Research Center at Indiana University. 

I had hypothesized that the supplemental classroom resource would cause greater 
gains in critical thinking skills, as measured through students' writing abilities, than 
in groups who either used the resource less effectively or didn't use the resource at 
all. The qualitative and quantitative results of the study were surprising and very 
interesting. 

The Results 

The first year's qualitative portion of the study yielded many interesting 
results that would have not been revealed had the study only measured the 
effects of the supplemental program quantitatively. 

The most important qualitative discovery for me was the use of the arts 
magazine by rural, and impoverished, students as a model of possibilities for 
themselves. Students in poor, remote areas were encouraged to go to technical 
or trade school beyond high school to secure a position that would allow them 
to secure a livelihood in their hometowns. The magazine offered models of possible 
alternatives to mundane employment. It was reported to me that some students took 
the risks of becoming artists after reading about other Indiana people who had 
successfully become artists despite economic realities and demands. 

Other students found that the art, writing, and poetry in the magazine 
served as models for them in their art and writing production. Students were 
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able to synthesize the products of the magazine into artistic efforts. 

Students who produced written journals, across subjects, often mentioned 
articles or art seen in the supplemental classroom resource. 

Since there were no requirements of use for the sponsored classroom 
resource there was a broad range of effective use of the resource. The usage 
varied from my finding boxes of unopened magazines in a school mail room 
and on a loading dock to the resource being valued and used as a primary 
classroom resource tool. I concluded that the magazine to student ratio 
was about one to two and one-half or three. In other words, about 7,000 
to 9,000 Indiana students were using the classroom resource in some manner 
each month. 

I was asked to co-author the study guide that accompanied the monthly 
magazine during the two year study. I found that the Program Manager of the 
classroom program had little experience in writing questions that would spawn 
higher level, and critical, thinking and that he had no experience with Fry Graphs 
for reading level analyses. This resulted in a monthly struggle of pulling the 
questions down from haughty levels to appropriate grade level reading levels and 
manipulating the questions to resemble the high end of Bloom's hierarchy. 

There was always a struggle to align the study guide with the intended purpose 
of the classroom resource. There was also ongoing dialogue about appropriate 
content for the school audience since the same publication was also sold on 
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newsstands and through the magazine’s circulation department. Occasionally, 
profanity would find its way into the magazine and I expressed concerns about 
such inclusions because my qualitative research led to discoveries of teachers 
ending up on Principal's carpets for profanity and nudity being included in the 
magazine. Time-pinched program teachers seldom had time to inspect the 
magazine cover-to-cover searching for inappropriate content but students took 
the time to find it and share it with their parents who, in turn, shared it with 
building Principals. 

The year-long quest of discovering the usage's of the resource allowed 
me to carefully group teachers into treatment groups for the quantitative 
portion of the study. In the quantitative portion, my observations made 
during the qualitative portion of the study were confirmed and reinforced. 

In my travels, I witnessed many teachers focusing on the basics in education. 
Ail teachers, for the most part, concentrated on studio production. The 
more effective ones had adopted a teaching technique called Discipline 
Based Art Education (DBAE). They incorporated art history, art criticism, 
and writing in art. Only a few teachers in other subject areas had shifted 
teaching strategies to include active student learning and few instructors 
encouraged creative and critical thinking. Most classes still concentrated on 
rote memorization and assessments that reinforced short, basic answers. 

Study results revealed that all treatment and control teacher groups actually 



7 



realized losses over the semester in their students' critical thinking abilities 
as measured through their writing (Appendix item I). Also, losses were evident 
among both female and male students (Appendix item II), among all student 
races (item III), across all school settings (Item IV), across all grade levels 
(item V), across all subjects (item VI), and across all state geographical zones 
(item VII). 

In the pretests, students were coming off summer vacations. Advanced 
students wrote wonderfully creative stories about inventions. They provided in- 
depth and lengthy descriptions. On the other hand, many students were unable to 
even describe an invention. By the end of the semester, the high-end students 
seemed to have been conditioned and reinforced to provide shorter and less detailed 
descriptions while the less able students were at least able to basically describe an 
invention. 

My observations in the first year of teachers teaching to the middle were 
confirmed by these findings. Results seemed to indicate that teachers were 
able to get lower performing students to rise to at least levels of minimal 
competency while more advanced students' creativity was not reinforced and 
their performance regressed, in an exaggerated fashion, to the mean. 

It should be noted that the teachers who were identified as the most 
effective users of the resource did have the least loss in students' measured 
critical thinking skills and there was statistical significance evident between 
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them and the blind control group and the less effective users of the resource. 
But, on the other hand, there were as many control teachers who had actual 
classwrde gains as there were treatment teachers with classwide gains. 

Only about ten percent of all the classrooms actually realized gains over the 

semester in classwide student critical thinking skills as measured through 
their writing abilities. 

So, while I had hypothesized that the supplemental resource would cause 

greater gains among its users, the reality was that the classroom resource, if 

used effectively, actually reduced classwide losses in students' critical thinking 

skills. I discovered universal losses in classwide student critical thinking skills, 

over the course of a semester, through student writing samples through this study. 

If these findings are valid then what can we do in our schools to get to the point to 

actually see gains in such areas? Since I did witness some of the teachers and 

classes that did have gains I have some knowledge about what it will take to 

have students get to the point of actually having universal class wide gains m 

critical thinking skills. To get to this point it will take a lot of work and a lot of 

change. Some people rush to conclude that its all about teacher effectiveness. 

That's too simple an explanation because it has to be more than just about 

teacher effectiveness if there is merit in providing supplemental resources in 
schools. 
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Conclusions 

The arts-based supplemental resource encouraged students to take career 
risks in the aits and encouraged them to be more creative in their artwork, 
performances, and writing. 

The absence of requirements of use, intensive program teacher training to 
familiarize teachers with program expectations, and the problems with alignment 
between the monthly study guide and the intended purposes of the resource 
impeded the effectiveness of it. 

With requirements of use in place coupled with intensive program teacher 
training and the delivery of grade-appropriate study guide questions designed to 
match the program's intent of increasing student critical thinking, resource usage 
could play an integral role in raising classwide critical thinking skills. 

Beyond the study findings, I had the opportunity to witness one person's attempts 
of basically funding and supporting this statewide educational venture. During the 
second year of the study the benefactress abruptly resigned and the classroom 
program, that has relied on soft money, is in jeopardy along with the publication 
itself. Witnessing the perils of relying on soft money for educational programs has 
caused me to conclude that state and federal funds need to be in place for such 
programs. The arts, and other educational resource ventures, should not rely on 
soft money. Money that is legislated into education is probably more secure than 
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soft money that can be pulled from programs for a justifiable reason or on a whim. 
Supplemental resource programs like this one should be funded in such a manner that 
ensures that the resource has enough time to be evaluated and improved or evaluated 
and modified or abolished based on its ineffectiveness. If supplemental curriculum 
programs have merit and worth then they should be funded in such a manner that they 
can carry on and make a difference in the educational lives of students. It is doubtful 
that this arts-based program will be in place much longer in Indiana schools and, despite 
its shortcomings, that could be addressed, and corrected, as a result of this evaluation, 
this is unfortunate. 
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A STUDY OF CRITICAL THINKING SKILLS AND WRITING ABILITY 

AMONG INDIANA STUDENTS 
Table 4.03 

Pretest and Posttest Means 
by Teacher Group 




Treatment Group One 



Treatment Group Two 



0 Pretest Mean 



Control Group One 



Control Group Two 



i Posttest Mean 



PRETEST 






T2 — Sih \ aaa\i — ucv .ooi8, Kange 4.00 

Cl - S d £ mS' v”™"" ' 4 ' 44, Sld Dev ' 6438 ' Ra "8' 5. 00 
C2 - S d £ 37 ' »• Sld Dev .6091 . Range 4.00 

Sid Err .0253, Vanance ,4297, Std Dev .6556, Range 4 00 



POSTTEST 



T1 — Std Err .0392, Variance .4346, Std Dev 
T2 — Std Err .0388, Variance .3154, Std Dev 
C 1 — S td Err .0329, Variance .27 1 1 , Std Dev 
C2 Std Err .0198, Variance .2626, Std Dev 



•6592, Range 4.00 
•5616, Range 3.00 
•5206, Range 4.00 
.5124, Range 3.00 
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A STUDY OF CRITICAL THINKING SKILLS AND WRITING ABILITY 

AMONG INDIANA STUDENTS 
Table 4.05 

Pretest and Posttest Means 
by Student Gender 




Female 

El Pretest Mean 



Male 

i Posttest Mean 



FEMALE 

PRETEST: Median 3.00, Std Err .0229, Variance .4033 
Skewness -.6925, S E Skew .0882, Kurtosis 1.66, S E Kurt .17, 
Range 4.00, Std Dev .6351 

POSTTEST: Median 2.00, Std Err .0198, Variance .3000 
Skewness -.2568, S E Skew .0882, Kurtosis .27, S E Kurt .17, 
Range 4.00, Std Dev .5477 



MALE 

PRETEST: Median 2.00, Std Err .0252, Variance .4087 
Skewness -.3847, S E Skew .0962, Kurtosis 1.77, S E Kurt .19, 
Range 5.00, Std Dev .6393 

POSTTEST: Median 2.00, Std Err .0226, Variance .3292 
Skewness -.0382, S E Skew .0962, Kurtosis 2.00, S E Kurt .19, 
Range 4.00, Std Dev .5737 
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A STUDY OF CRITICAL THINKING SKILLS AND WRITING ABILITY 

AMONG INDIANA STUDENTS 
Table 4.06 

Pretest and Posttest Means 
by Student Race 



Score 

5 



4 



(-0.25) 



(-0.04) 



(-0.10) (-0.46) (-0.16) 



(*0.17) 



3 



2.56 




White 



3.18 




Black 



Hispanic 



Asian Native Amer. Other 



□ Pretest Mean 



■ Posttest Mean 






WHITE 


BLACK 


HISPANIC 


ASIAN 


NATIVE AM. 


OTHER 




PRE/POST 


PRE/POST 


PRE/POST 


PRE/POST 


PRE/POST 


PRE/POST 


Std Dev 


.6505/.5586 


.5667/.5012 


.5258/5763 


.7508/.7862 


.8165/.8367 


.7048/.6077 


Variance 


.423 1/.3 1 20 


.3212/.2512 


.2764/3322 


.5636/.6182 


.6667/.7000 


.4967/3693 


Std Err 


.01 84/.01 58 


.061 1/.0540 


.081 1/.0889 


.2264/.2371 


.3333/3416 


. 1 66 1/. 1432 


Range 


5.00/4.00 


4.00/3.00 


2.00/3.00 


2.00/2.00 


2.00/2.00 


3.00/2.00 
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A STUDY OF CRITICAL THINKING SKILLS AND WRITING ABILITY 

AMONG INDIANA STUDENTS 
Table 4.07 

Pretest and Posttest Means 
by School Setting 



Score 

5 




2.37 



Advantaged Urban Small City/Town 

B Pretest Mean 



(-0.23) 



(-0.24) 




Disadvantaged Urban Rural 

■ Posttest Mean 



PRETEST 




Adv. Urban 


Std Err 


Small City 


Std Err 


Dis. Urban 


Std Err 


Rural 


Std Err 



.0476, Variance .4627, 
.0390, Vatiance .4170, 
.029 1 , Variance .452 1 , 
.0297, Variance .3537, 



Std Dev .6802, Range 4.00 
Std Dev .6458, Range 5.00 
Std Dev .6724, Range 4.00 
Std Dev .5947, Range 4.00 



POSTTEST 
Adv. Urban 
Small City 
Dis. Urban 
Rural 



Std Err .0450, Variance .4122, Std Dev .6421, Range 4.00 
Std Err .0330, Vatiance .2975, Std Dev .5454, Range 4.00 
Std Err .0229, Variance .2817, Std Dev .5307, Range 4.00 
Std Err .0278, Variance .3091, Std Dev .5560, Range 4.00 




17 



V 



A STUDY OF CRITICAL THINKING SKILLS AND WRITING ABILITY 

AMONG INDIANA STUDENTS 
Table 4.08 

Pretest and Posttest Means 
by Student Grade Level 

Score 



4 



(-0.17) (-0.10) (-0.05) 



(-0.26) 



(-0.15) (-0.26) 



(-0.24) (-0.09) 



2.78 




□ Pretest Mean ■ Posttest Mean 



GRADE 


FIVE 


SIX 


SEVEN 


EIGHT 


NINE 


TEN 


ELEVEN 


TWELVE 


PRETEST 


Std Err 


.1085 


.0952 


.0721 


.0551 


.1043 


.0259 


.0599 


.0483 


Variance 


.2826 


.3720 


.3068 


.3547 


.2934 


.4539 


.3122 


.4323 


Std Dev 


.5316 


.6099 


.5539 


.5956 


.5417 


.6737 


.5588 


.6575 


Range 


2.00 


3.00 


3.00 


4.00 


2.00 


4.00 


2.00 


4.00 


POSTTEST 


Std Err 


.0576 


.0654 


.0659 


.0476 


.0697 


.0199 


.0709 


.0479 


Variance 


.0797 


.1756 


.2566 


.2654 


.1311 


.2686 


.4373 


.4236 


Std Dev 


.2823 


.4149 


.5065 


.5152 


.3620 


.5182' 


.6613 


.6509 


Range 


1.00 


1.00 


2.00 


3.00 


1.00 


4.00 


4.00 


3.00 
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A STUDY OF CRITICAL THINKING SKILLS AND WRITING ABILITY 

AMONG INDIANA STUDENTS 
Table 4.09 

Pretest and Posttest Means 
by School Subject 



Score 




0 Pretest ■ Posttest 



SUBJECT 


Elem. 


J.H. Art 


J.H. L. Arts 


J H Other 


H.S. Art 


H.S. L. Arts 


H.S. S.S. 


H.S. Other 


PRETEST 
Std Err 
Variance 
Std Dev 
Range 


.0950 

.2796 

.5287 

2.00 


.0464 

.3398 

.5829 

4.00 


.1439 

.2692 

.5189 

1.00 


.1060 

.4386 

.6623 

4.00 


.0449 

.3754 

.6127 

3.00 


.0216 

.4349 

.6595 

4.00 


.1181 

.5583 

.7472 

4.00 


.1542 

.4044 

.6359 

3.00 


POSTTEST 
Std Err 
Variance 
Std Dev 
Range 


.0672 

.1398 

.3739 

1.00 


.0407 

.2613 

.5111 

3.00 


.1404 

.2564 

.5064 

1.00 


.0614 

.1471 

.3835 

2.00 


.0421 

.3304 

.5748 

3.00 


.0185 

.3180 

.5639 

4.00 


.1071 

.4590 

.6775 

3.00 


.1664 

.4706 

.6860 

2.00 
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A STUDY OF CRITICAL THINKING SKILLS AND WRITING ABILITY 

AMONG INDIANA STUDENTS 
Table 4.10 

Pretest and Posttest Means 
by State Zone 



Score 

5 



2.69 



(-0.06) 


(-0.12) 


(-0.34) 


Ull62 


2.47 


171 




me S.W. 


S.E. S.W.C. 

□ Pretest Mean 


E.C. 


N.W.C. 


N.W. 


N.C. N.E. 

■ Posttest Mean 




M.C. 




S.W. 


S.E. 


S.W.C. 


E.C. 


N.W.C. 


N.W. 


N.C. 


N.E. 


M.C. 


PRETEST 


Std Err 


.0716 


.0267 


.0722 


.0591 


.0693 


.0479 


.0688 


.0592 


.0612 


Variance 


.4662 


.4482 


.2189 


.4091 


.3985 


.3653 


.3126 


.3187 


.5140 


Std Dev 


.6828 


.6695 


.4679 


.6396 


.6312 


.6044 


.5591 


.5645 


.7169 


Range 


4.00 


5.00 


1.00 


4.00 


3.00 


3.00 


2.00 


3.00 


4.00 


POSTTEST 


Std Err 


.0650 


.0205 


.1036 


.0454 


.0545 


.0449 


.0771 


.0571 


.0578 


Variance 


.3841 


.2642 


.4506 


.2414 


.2465 


.3205 


.3928 


.2972 


.4578 


Std Dev 


.6198 


.5140 


.6713 


.4913 


.4965 


.5661 


.6267 


.5452 


.6766 


Range 


3.00 


4.00 


3.00 


2.00 


2.00 


3.00 


3.00 


3.00 


4.00 
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